Skip to content

multikernel: support isolated SR-IOV VF assignment - #4

Draft
nickolaev wants to merge 9 commits into
multikernel:masterfrom
nickolaev:sriov-vf-assignment
Draft

multikernel: support isolated SR-IOV VF assignment#4
nickolaev wants to merge 9 commits into
multikernel:masterfrom
nickolaev:sriov-vf-assignment

Conversation

@nickolaev

@nickolaev nickolaev commented Jul 22, 2026

Copy link
Copy Markdown

Multikernel instances currently lack a complete PCI ownership model. A spawned
x86 kernel needs enough platform and device state to enumerate and operate its
assigned device without exposing or disturbing devices retained by the primary
kernel.

This series adds end-to-end assignment of an SR-IOV VF to a spawned kernel while
the primary retains the PF and enforces DMA isolation.

The series:

  1. Adds PCI identity, BAR resources, and host-bridge information to each Multikernel instance.
  2. Transfers and restores the x86 ECAM configuration required by the spawned kernel.
  3. Exposes assigned devices through synthetic PCI roots instead of enumerating shared physical bridges.
  4. Filters configuration-space accesses before hardware access so the spawned kernel cannot probe or modify unassigned functions.
  5. Adds exclusive SR-IOV VF leases while keeping the PF bound to its host driver.
  6. Makes memory, CPU, PCI, and host-bridge reservation atomic, with reverse-order unwind on failure.
  7. Creates a host-owned IOMMU paging domain for each assigned VF and maps only memory owned by the instance.
  8. Quiesces and resets the VF before release, then restores its original host driver.
  9. Fails closed on IOMMU faults, unexpected device removal, or loss of the VF assignment.
  10. Uses the existing x86 Multikernel platform quirk for spawned-kernel timer initialization.

The resulting ownership model is:

  • the primary kernel retains the PF and controls the VF assignment lifecycle;
  • only SR-IOV VFs can be assigned;
  • each VF is exclusively owned by one Multikernel instance;
  • the primary owns the VF's IOMMU domain and DMA mappings;
  • only instance-owned memory is reachable through DMA;
  • the secondary sees its assigned VF through a synthetic PCI root;
  • the PF, physical bridges, and unrelated PCI functions are not exposed;
  • configuration accesses to unassigned functions do not reach hardware;
  • VF identity and BAR resources survive the handoff through owned instance metadata; and
  • teardown stops DMA before returning the VF to the host.

Assignment requires an active hardware IOMMU, a singleton IOMMU group, isolated MSI delivery, and a DMA aperture
capable of covering every instance memory region. Instance memory cannot be changed while an IOMMU-backed VF lease is active.

This work can be demonstrated using the main branch here: https://github.com/nickolaev/multikernel

The QEMU harness verifies VF enumeration and traffic in the spawned kernel, continued PF operation in the primary, IOMMU containment,
exclusive ownership, hostile lifecycle operations, repeated create/delete cycles, driver restoration, and fail-closed handling of unexpected VF loss.

It has been tested with qemu-system-x86_64 using Q35, IGB SR-IOV, Intel IOMMU, interrupt remapping, and strict IOMMU mode. It has not yet been tested on real
SR-IOV hardware.

@congwang-mk

Copy link
Copy Markdown

Hi @nickolaev

Nice work, thanks for the PR! I finally have time to look into it.

A few high-level comments:

  1. If I understand it correctly, it only works with IOMMU off? This looks a big limitation.
  2. VF lifetime is coupled to the PF driver with no coordination
  3. Exposed bridges are shared and mutable?
  4. For the timer fix, it would be better if we could use existing x86 quirks.

@nickolaev

Copy link
Copy Markdown
Author

@congwang-mk thanks for the high-level review. Indeed, this is a naive implementation of "let's make ICMP pass"; It is very fragile as it is. The questions you asked are setting a proper direction. I have some ideas; let me try to clean things up and will ping you once I have a better patch series.

@nickolaev
nickolaev force-pushed the sriov-vf-assignment branch 2 times, most recently from 8d8960b to 4e05ab7 Compare July 30, 2026 16:41
@nickolaev

Copy link
Copy Markdown
Author

@congwang-mk I have updated the commit history with the revised work and the PR message reflects the new state.
Please take a look when you can.

@nickolaev
nickolaev force-pushed the sriov-vf-assignment branch from 4e05ab7 to f5c1493 Compare July 30, 2026 18:02
A spawned kernel cannot safely rediscover the resource layout of
a device that remains physically attached to the host. Make the
instance description the owned source of PCI identity, BAR, and
host-bridge metadata.

Snapshot device resources and ECAM descriptors into the Multikernel
device-tree transport. Parse, clone, and release that data with the
instance lifecycle, and restore it through KHO without retaining live
host PCI objects.

This establishes the transport and ownership model only; later patches
perform the exclusive VF lease.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
PCI ECAM discovery and restoration are architecture policy. Keeping that
policy in generic Multikernel or PCI probe code makes the transport
depend on x86 implementation details.

Provide a locked MMCONFIG region iterator, snapshot only windows
that cover assigned devices, and restore those windows from the
instance description during x86 platform setup. Select the restored
ECAM operations before the spawned kernel scans PCI.

Generic MMCONFIG code remains unaware of Multikernel instances.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
A spawned kernel must not probe or mutate functions retained by the
primary kernel. Filtering in the generic PCI probe path is too late
and places Multikernel policy in shared PCI code.

Wrap the x86 configuration-space operations so unassigned functions
are rejected before hardware access. Present the assigned identity
from instance metadata, retain only bridge traversal needed to reach
an assignment, and restore the recorded BAR resources during early fixup.

This keeps assignment policy in Multikernel and removes the special
case from drivers/pci/probe.c.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Moving a PCI inventory entry between instances does not coordinate
with the live host device or prevent two instances from requesting
the same function.

Introduce an assignment object for each leased device and serialize
lease creation and release. Validate the root inventory against the
live PCI function, accept only SR-IOV VFs, preserve the original
host-driver state, move the inventory only after the lease commits,
and use PCI bus notifications to fail an instance if its PF or VF
disappears unexpectedly.

The PF stays bound to its host driver. Only the VF changes ownership,
and instance teardown returns it to the root inventory.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Instance creation reserves memory, CPUs, PCI devices, host-bridge
metadata, and platform devices. Returning an error after any one of
those transfers can otherwise expose a partially populated instance
and leak resources from the root.

Validate configuration counts and lists before transfer, acquire each
resource class in a fixed order, and unwind every completed step in
reverse order. Centralize release so create failure, instance deletion,
and final reference teardown share the same all-or-nothing semantics.

Balance references acquired for remote memory add and remove operations
on every success and error path so resource hotplug cannot pin a
deleted instance.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Passing physical host bridges to a spawned kernel leaves bridge
configuration shared and mutable even when endpoint probing is
filtered. The primary and secondary could then program the same
routing state independently.

Treat ECAM descriptors as discovery metadata only. Create a synthetic
root for each required segment and bus range, scan assigned endpoints
through the filtered configuration operations, and never enumerate the
physical bridge functions in the spawned kernel.

Fail the spawned-kernel PCI initialization when root metadata is
missing, overlapping, or cannot be mapped so a partial topology is
never exposed.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
A config-space allowlist does not isolate DMA. Letting a leased VF
retain the host default domain would allow it to reach memory outside
the spawned instance, while disabling the IOMMU is not an acceptable
assignment model.

Require IOMMU support for VF leases and accept only singleton groups
with isolated MSI delivery and compatible reserved regions. Build
a host-owned paging domain, map only the instance memory regions
with their allowed permissions, bind the VF to a driver-managed-DMA
assignment driver, and attach the group before committing the lease.

Any validation, mapping, binding, or attach failure unwinds the
domain and leaves the VF with its host driver. Balance target-instance
references when IOMMU state rejects memory changes. The spawned kernel
receives no ownership of the host IOMMU programming.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Returning a VF while it can still issue DMA races the IOMMU teardown
and host-driver reprobe. Unexpected PF or VF removal must also stop
an active instance instead of silently losing its assigned device.

Clear bus mastering, wait for pending transactions, and issue
function-level reset while the assignment domain is still attached.
Then detach and free the domain, restore the saved driver override
and host driver, and only afterwards return the inventory to the root.

Lease-loss notifications force an active instance to halt and mark it
failed. Rollback entries that were prepared but never committed skip
device quiesce and driver restoration, releasing only their prepared
IOMMU resources.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Spawn kernels do not own the PIT, PIC, or IO-APIC resources. The
X86_SUBARCH_MULTIKERNEL platform quirk therefore leaves timer_init
and wallclock_init as no-ops so an instance never programs host timer
hardware or requests IRQ0.

setup_percpu_clockev() initializes the local APIC timer for instance
ticks. Keeping global_clock_event unset bypasses LAPIC timer verification,
whose fallback path requires the unavailable legacy IRQ0.

Document that timer path next to the platform quirk.

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
@nickolaev
nickolaev force-pushed the sriov-vf-assignment branch from f5c1493 to b2d1b55 Compare July 30, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants